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WITH  PERIODIC  REVIEW 

By 
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August,  1969 

Chairman :   Michael  E.  Thomas 

Major  Department:   Industrial  and  Systems  Engineering 

The  general  problem  of  finding  optimal  policies  for  oper- 
ating queuing  system.s  is  investigated  in  order  to  determine 
when  to  open  or  close  servers  so  as  to  minimize  cost.   The 
queue  size  is  observed  at  equally  spaced  intervals  of  time 
and  the  structure  of  the  optimal  policy  is  determined. 

The  main  restriction  made,  at  first,  is  to  limit  the  model 
to  a  single  server  system.   Justification  for  a  form  of  an  opti- 
mal policy  is  given.   Then  the  proofs  that  this  policy  is  in 
fact  optimal  for  the  finite  horizon,  infinite  horizon,  and 
averaging  cost  criterion  are  given.   Also,  conditions  for 
determining  that  the  server  should  never  be  closed  with  custo- 
mers in  queue  are  presented. 

Two  computational  schemes  for  determining  the  optimal 
policy  are  presented.   The  first,  utiliziiig  system  parameters, 
minimizes  average  cost  by  computing  the  expected  waiting  time. 
The  other  solves  the  infinite  horizon  discounted  cost  criterion 
by  direct  use  of  Howard's  Markov  programming  algorithm. 


The  problem  is  then  generalized  to  allow  for  a  fixed  num- 
ber of  servers.   Most  of  the  results  from  the  single  server 
follow  immediately.   The  main  difficulty  is,  therefore,  to 
establish  and  justify  the  desired  rule  so  that  it  is  analo- 
gous with  the  single  server  and  is  still  as  general  as  possible. 

The  primary  mathematical  concept  used  in  solving  the  prob- 
lem is  dynamic  programming.   This  technique  yields  the  solution 
to  the  finite  and  infinite  discounted  cost  criteria  in  both 
models  presented.   Some  of  the  results  from  replacement  theory 
help  in  solving  the  averaging  case. 

Finally,  the  conclusion  shows  how  this  work  is  an  extension 
of  the  related  work  in  the  field,  and  points  out  several  exten- 
sions which  naturally  follow.   The  fact  that  the  form  of  the 
optimal  policy  is  both  intuitively  appealing  and  easy  to  imple- 
ment makes  the  results  useful  in  many  queuing  and  related 
systems,  such  as  inventory  and  replacement  models. 
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CHAPTER  I 

INTRODUCTION  AND  PROBLEM  STATEMENT 

Introduction 

An  important  problem  to  consider  is  the  economic  behavior 
of  a  stochastic  service  system.   Most  research  in  queuing 
analysis  has  dealt  v/ith  the  underlying  probabilistic  structure 
rather  than  identifying  optimal  policies  for  the  operation 
of  the  system.   Within  the  operation  of  a  service  station 
there  is  a  fraction  of  time  in  which  the  server  is  idle.   This 
idle  fraction  can  be  eliminated  in  one  of  two  ways.   Firsc, 
the  server  can  carry  out  auxiliary  duties  during  this  idle 
fraction  or,  as  an  alternative,  there  can  be  a  partial  or 
complete  elimination  of  the  station.   We  are  concerned  with 
the  latter,  i.e.,  when  to  open  or  close  our  station  in  order 
that  some  objective  is  optimized.   The  solution  to  this  prob- 
lem is  by  no  means  immediately  obvious , especially  when  there 
is  a  statistical  input  of  customers.   In  this  paper  we  will 
discuss  a  reasonable  form  for  an  optimal  solution,  prove 
under  what  conditions  this  solution  exists  and  discuss  exten- 
sions including  computational  refinements  for  this  type  of 
solution. 

Previous  Results 

The  justification  for  this  work  stems  from  the  similarity 


between  queuing  and  inventory  models.   Much  work  has  been 
done  using  explicit  cost  functions  on  inventory  modexs 
in  particular  getting  optimal  policies  of  a  particular  struc- 
ture, such  as  the  s-S  policy.   Because  of  this  similarity  in 
structure  it  seemed  reasonable  to  attempt  to  fit  explicit 
cost  functions  to  the  model  with  the  hope  of  finding  opti- 
mal policies  similar  in  structure  to  the  s-S  policy. 

The  m.athematical  background  for  this  dissertation  comes 

2  n 

first  from  dynamic  programming  '    and  then  from  many  papers 
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on  Markovian  Decision  Processes.  '    Many  of  these  works  are 

discussed  in  a  survey  paper  by  the  author.     In  a  paper  on 
replacement  policies  which  is  an  application  of  sequential 
decision  process,  Derman   defines  a  class  of  rules  as  con- 
trol limit  rules.   The  form  of  this  rule  was  helpful  in  ob- 
taining optimal  rules  for  the  queuing  model. 

The  earliest  paper  on  optimization  of  queuing  systems 
was  by  Edie.    Ke  uses  optimization  techniques  in  determin- 
ing the  best  level  of  service  based  on  traffic  volume  at  toll 
booths.   Then  the  number  of  toll  booths  required  at  any  time 
of  day  can  be  established.   Edie ' s  results  provided  a  minimum 
number  of  toll  collectors  providing  acceptable  service  with 

relief  for  the  toll  collectors. 

17  •    •      '^ 

A  paper  by  Yadin  and  Naor   was  an  important  inspira- 
tion for  this  work.   They  dealt  with  the  partial  elimination 
of  a  single  service  station  under  a  (0,R)  doctrine.   This  is 
where  the  server  when  closed  remains  closed  until  R  customers 
are  present  in  the  system  and  remains  open  until  the  system 


is  empty.   They  give  properties  of  this  system  under  this 
rule  and  also  give  computational  aspects  leading  to  a  closed 

form  solution  for  the  value  of  R. 
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Most  recently  Heyman   and  Sobel    have  recognized  the 

importance  of  explicit  cost  functions  for  stochastic  service 
systems.   Heyman  considers  a  single  channel  queue  with  Poisson 
arrivals  and  arbitrary  service  time  distribution.   Upon  com- 
pletion of  arrival  and  service  epochs  he  shows  that  for  infi- 
nite horizon  models  there  exists  an  optimal  stationary  policy 
of  particular  form.   He  also  finds  an  explicit  cost  function 
and  thereby  is  able  to  find  an  optimal  policy  as  a  function 
of  the  parameters  of  the  system.   Heyman  next  considers  a 
two-channel  model,  one  of  which  is  always  open,    with  the 
added  assumption  that  the  service  time  distribution  is  expo- 
nential.  He  again  shows  what  form  the  optimal  policy  m.ust 
take  and  uses  Jewell's  Markov  Renewal  Programming  technique 
to  find  the  policy. 

Sobel  generalizes  the  problem  by  making  the  arrival  dis- 
tribution arbitrary.   Also,  where  Heyman  uses  a  linear  hold- 
ing cost,  Sobel  uses  a  very  general  holding  cost.   He  then 
shows  existence  for  the  same  type  of  optimal  policy  as  does 
Heyman . 

General  Problem  Statement 

Consider  "customers"  arriving  at  a  service  facility  from 
an  infinite  population  according  to  some  statistically  steady 
input  stream  with  distribution  function  A(t).   Also,  assume 


there  are  at  most  s  servers,  each  with  distribution  of  ser- 
vice tine  B,  (t) ,  k=l,  2,  ...  ,  s.   The  system  is  observed 
at  discrete  times  (t, ,  t-  . . . ) .   We  are  interested  in  first 
showing  the  existence  and  then  finding  a  policy  of  form  (A) 
so  as  to  minimize  some  cost  structure^  where  form  (A)  is: 
(A)  for  each  k  (k  =  1,  2,  ...  ,  s)  identify  a  unique 
ordered  pair  [a* (t) ,  b*(t)]  such  that  if  i,  the  number 
in  the  system,  falls  between  a*  (t)  and  b*  (t)  a  specific 
action  will  be  determined.   If  i  does  not  fall  between 
these  values  then  no  action  will  be  taken.   By  an  action 
v/e  mean  how  many  servers  should  be  opened  or  closed  . 
The  form  of  the  optimal  policy  for  each  specific  prob- 
lem will  be  of  this  structure  and  will  also  inherently 
make  it  clear  how  many  servers  should  be  opened  or  closed. 
This  differs  from  the  aforementioned  works  because  of  the 
following  refinements: 

(1)  Assume  exponential  inter-arrival  and  service  distri- 
butions . 

(2)  Allow  for  a  fixed  and  finite   number  of  servers. 

(3)  Show  the  existence  of  an  optimal  policy  for  finite 
horizon,  infinite  horizon  and  averaging  cost  criteria. 

(4)  Make  the  decision  points  at  the  end  of  time  periods 
of  equal  length.   This  is  a  major  difference  between 
this  work  and  the  others.   Observing  at  the  end  of 
equal  time  intervals  causes  many  difficulties  that 
are  not  encountered  when  observation  points  are  the 
completion  of  arrival  or  service  epochs.   These  will 
be  pointed  our  later  in  the  text. 


CHAPTER  II 

SINGLE  SERVER  MODEL 

Let  US  first  consider  a  modification  of  this  problem 
which  is  far  more  tractable. 

A  Specific  Problem 

As  a  specific  problem  let  us  make  the  following  changes 
from  the  general  problem  formulation. 

(I)  Assume  there  is  just  one  server.   This  is  done 
solely  to  simplify  the  problem, and  hopefully  will  give  insight 
to  the  multiserver  case. 

(II)  Assume  the  system  has  no  bounds  on  capacity.   But 
also  assume  there  is  only  a  finite  number  of  states  for  v;hich 
decisions  will  be  made,  i.e.  when  observed  with  more  than  M 
customers,  for  some  finite  M,  the  decision  will  always  be  to 
open  if  we  are  closed.   This  is  not  a  major  restriction  and 
the  reason  for  it  will  be  pointed  out  later  in  the  text. 

(III)  Assume  stationary  solutions.   Blackwell   assures 
that  both  a*  and  b*  will  not  be  functions  of  time,  in  an 
infinite  horizon  model,  because  of  the  finiteness  of  the 
action  space  (here,  only  two  possible  decisions). 

(IV)  Assume  both  the  times  between  arrivals  and  services 
are  distributed  exponentially.   Some  coiriinents  are  needed  to 


justify  this  assumption.   A  desirable  characteristic  of  a 
real-life  service  system  is  that  the  monitoring  or  checking 
times  are  not  too  frequent.   In  fact,  we  would  like  to  ob- 
serve our  system  at  equal  time  intervals  (once  a  minute,  hour, 
day,  etc.).   To  ensure  the  problem  is  solvable  the  sequence 
of  states  must  form  a  Markov  Chain  (number  of  customers  at 
our  service  station  must  be  independent  of  the  past  history 
of  the  process);  that  is,  there  must  be  stationary  transition 
probabilities.   This  implies  that  the  number  of  arrivals  and 
services  must  only  depend  on  the  length  of  the  time  period. 
In  order  not  to  increase  the  dimensionality  cf  the  state  space, 
A(t)  and  B(t)  are  restricted  to  be  exponential  distribution 
functions.   If,  for  example,  B(t)  has  a  general  probability 
distribution,  to  ensure  the  Markovian  property  it  would  be 
necessary  to  observe  the  system  at  the  completion  of  each 
service  when  it  was  open.   We  have  chosen  to  be  more  general 
in  our  observation  points  than  in  the  arrival  and  service 
distributions.   However,  much  of  the  analysis  and  many  of  the 
results  can  be  used  for  the  other,  more  general,  case. 

The  specific  problem  can  now  be  stated.   Customers  arrive 
before  a  single  server  according  to  the  Poisson  probability 
law.   The  time  between  services  is  exponential.   The  system 
is  observed  at  equal  time  intervals  t  =  1,  2,  ...  We  are 
interested  in  first  shov/ing  the  existence  and  then  finding 
a  policy  of  form  (A')  so  as  to  minimize  some  cost  function; 
where  (A')  is: 

{A')  identify  a  unique  pair  (j*,  i*)  with  the  interpre- 
tation that  if  and  only  if  the  server  is  open  when 


observed  and  there  are  at  most  j*  customers  present  will 
the  system  be  closed  and  if  and  only  if  the  server  is 
closed  and  there  are  at  least  i*  customers  present  when 
observed  will  the  server  be  opened. 
Note  that  this  policy  could  be  expressed  precisely  in  the  form 
of  (A)  from  Chapter  I.   If  for  k  =  0,  no  servers  open,  we 
choose  (a*,  b*)  to  be(i*,  <")  then  the  action  will  be  to  open 
the  server  if  the  number  in  the  system  is  between  a*  and  b*, 
and  no  action  would  be  taken  otherwise.   Similarly,  if  k  =  1 
and  (a*,  b*)  is  chosen  as  (-«,  j*)  then  the  server  will  only 
be  closed  when  the  number  of  customers  is  between  a*  and  b*. 

This  differs  with  Yadin  and  Naor  in  that  they  assumed 
a  (0,R)  doctrine  and  computationally  found  the  optimal  value 
of  R. 

Characteristics 

The  properties  of  this  model  can  now  be  given  as  follows: 

(a)  The  Arrival  Pattern  —  Customers  arrive  so  that  the 
time  between  successive  arrivals  is  distributed  exponentially. 

(b)  The  Service  Mechanism  —  Customers  are  served  in 
order  of  their  arrival  so  that  the  time  between  successive 
services  is  distributed  exponentially. 

(c)  Cost  Structure  -- 

(i)     A  -  Constant  cost  of  opening  a  closed  server 
(ii)    B  -  Constant  cost  of  closing  an  open  server 
(iii)   C  -  Cost  of  operating  the  server  for  one  time 

period 
(iv)    K(i)  -  Holding  cost,  i.e.,  cost  incurred  when 
i  customers  are  observed  in  the  system. 


8 

These  costs  may  or  may  not  be  discounted  with  discount 
factor  a. 

The  decision  process  is  as  follows.   The  system  is  observed 
at  times  t  =  1,  2,  ...  and  the  state  of  the  system  X   is  noted. 
X  =  i  implies  there  are  i  customers  in  the  system  at  time  t. 
The  decision  is  then  made  to  either  leave  the  server  in  its 
present  state  (opened  or  closed)  or  to  open  a  closed  server  or 
to  close  an  open  server,  or  if  X  >  M  and  the  server  is  closed  it 
must  be  opened. 

The  sequence  of  states  {X.  }  forms  a  Markov  Chain  and  changes 
of  states  occur  according  to  the  following  probabilistic  struc- 
ture, 

p. .  =  Pr   {X^.,  =  j I X^  =  i  and  the  server  is  closed} 
'^ij         t+1    -■  '  t 

q.  .  =  Pr   {X^  ■,  =  J  I X^  =  i  and  the  server  is  open} 
^ij         t+1    -•  '  t 

Existence  of  a  Control  Limit  Rule 

Consider  a  single  channel  system  with  the  aforementioned 
characteristics.   We  would  like  to  establish  the  existence  of 
a  control  limit  rule.   This  is,  if  the  server  is  closed  show 
that  there  exists  a  unique  numiier  i*,  such  that  if  the  number 
of  customers  observed  in  the  system  is  no  less  than  i*  then 
open  the  server.   Further,  if  the  number  observed  is  less 
than  i*  then  keep  the  server  closed.   This  rule  has  intuitive 
appeal,  saying,  that  if  the  server  is  not  open  with,  say,  7 
customers  in  queue  then  certainly  it  will  not  be  open  with, 
say,  5. 


It  is  desirable  to  show  the  existence  of  this  number  for 
an  averaiging  objective  function.  In  doing  so  we  obtain  simi- 
lar results  for  both  the  discounted  finite  and  infinite  hori- 
zon models. 

Dynamic  Programming  Formulation 

Consider  the  solution  to  the  finite  horizon  model  using 
dynamic  programming.   Define, 

({)   (i,a)  =  minimum  expected  cost  with  n  time  periods 
remaining  when  i  customers  are  observed  in 
the  queue  using  discount  factor  a,,  and  k  =  0, 
1  depending  on  whether  the  server  is  closed 
or  open  respectively. 
Then  the  recursive  equations  are 


(j>°  {i,a)  =  min  f  K(i)  +  aZ  p..  (j)   ,  (j,a) 
n  I         j^Q   ij   n  X 


K(i)  +  A  +  C  +  a  I   q^.  4ij^_3_  ( j  ,  a) 

where  the  alternatives  are  to  keep  the  server  closed  or  to  open 
it. 


1  t 

And    (J>   (x,a)  =  mm 


00  > 

K(i)  +  B  +  a  I   p.  .  <t)   ,  (j,a) 

j  =  0  ^J  "  -^ 


K(i)  +  C  +  a  E  q.  .  ^^-1^^'"^  > 
j  =  0 

with  alternatives  closing  the  server  or  keeping  it  open,  where 


it  is  assumed  that 


({."  (m,a)  =  K(m)  +  A  +  C  +  I 


n 


j  =  0 


q  .  '^   ,  (j  ,a  )  for  m  >  M 
^m]  ^m-1^  — 
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Define, 


(J,Q(i,a)  =  K(i)  ,  k  =  0,  1 

i.e. ,  the  terminal  cost  of  our  process  is  the  holding  cost 
for  the  number  of  customers  observed.   In  addition,  assume 
(i)   that  K(i)  is  a  nondecreasing  function  of  i ,  a 

reasonable  assumption  and  one  often  made  in  inven- 
tory or  replacement  analysis. 

eo 

and  (ii)   that  for  any  nondecreasing  function  h{j),   Zp. ■h(j) 

3  =  0  ^3 

00 

and   Iq.  .h(j)  are  nondecreasing  functions  of  i. 
j=0  ^3 

This  says,  given  some  subset  of  states  K  =  {k, 
k+1,  ...}  ,  the  Chance  of  entering  K  is  nondecreasing 
in  i. 
Consider  the  server  closed.   For  n  =  1  we  have 


(ji,  (i,a)  =  min 


K(i)  +a  Z  p.  .  K(j) 
j  =  0  ^3 


l. 


K(i)  +  A  +  C  +a  Z  q.  .  K(j) 

j  =  0  ^3 

From  assumptions  (i)  and  (ii)  ws  know  both  of  these  alternatives 
and  also,  ^^   (i,a)  are  nondecreasing.   If  at  i  =  0,  the  second 
alternative  is  chosen  then  we  must  ensure  it  must  be  chosen 
for  all  i  (always  open  server) .   If  at  i  =  0  the  first  alterna- 
tive is  chosen  then  these  two  nondecreasing  curves  may  cross  in 
at  most  one  place,  the  i*  being  sought.   If  they  don't  cross 
at  all  before  M  then  the  server  will  never  be  opened  until 
at  least  M  customers  are  present.   The  possibilities  are 
illustrated  in  Figure  1.   To  ensure  case  c  cannot  occur  an 
additional  assumption  must  be  made. 


COST 
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COST 


y 


COST 


COST 


a  -  Never  open  server 

c  -  Undesirable  case 

-  -  -  -  cost  for  closed  server 

cost  for  open  server 


d   i*  i 

b  -  Never  close   server 
d  -  Unique  opening  point 


Figure  1:   Associated  costs  for  two  alternatives 
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We  should  comment  here  how  Heyman's  model,  that  of  check- 
ing after  each  arrival , fails  to  encounter  this  difficulty. 
Let  us  say  that  case  c  denotes  the  graphs  of  the  curves  in 
Equation  1.   Let  i,  be  the  first  crossing  point  and  i2(>i-j_) 
the  second,  that  is  keep  the  server  closed  for  i  <i^  and 
i  >_i2   and  open  otherwise.   If  the  system  is  observed  at  i  <ij_ 
and  then  at  each  arrival  thereafter,  as  Heyman  does,  as  soon 
as  i,  is  reached  the  server  will  be  opened  and  the  state  i  >_  i2 
for  a  closed  server  will  never  be  reached.   Checking  at  dis- 
crete time  points,  however,  may  result  for  some  t  in 

X.  =  i   for   i  <  i^ 
and  X^_j_^  =  j    for    J  ^  ^2 

resulting  in  keeping  the  server  closed  at  both  points,  and 
hence  his  rule  of  always  opening  after  i,  may  not  hold.  - 

Additional  Restriction 

As  so  often  happens  in  this  type  of  problem  when  a  barrier 
is  reached,  an  additional  condition  must  be  imposed.   By  anal- 
ogy to  the  definition  of  convexity  we  im.pose  the  following 
condition: 

eK(j)  +  (1-e)  K(i)  >_  K(ej  +  (l-9)i)  for 

ee[0,l]  and  the  argument  for  K(.)  integral     (2) 
For  obvious  reasons  we  call  (2)  the  convex  property.   It  would 
be  possible  to  achieve  the  same  results  by  defining  the  envelope 
of  the  function  as  convex.   However,  there  is  very  little  addi- 
tional computation  in  analyzing  the  discrete  function. 
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Uniqueness  Proof  for  n  =  1 
Recalling  that 


()),  (x,a)  =  mm 


K(i)  +  a  Z  D.  .K(j) 
j  =  0  ^^ 


K(i)  +A  +  C  +  aZq.  .K(j) 

j  =  0  ^^ 


it  must  be  determined  when 


K(i)  +  aIp..K(j)  (-)  K(i)  +  A  +  C  +  aIq..K(j) 
j  =  0  -^J       _  -i=n  ^3 


j=0 


or 


I  K{j)  [p.  .  -  q.  .]  (-)  ^-^LA  k    A 
j  =  0       ^3    ^3    -     " 

This  will  be  done  by  showing  that 

CO 

^      K(j)   [p.  .  -  q.  .] 

is  a  nondecreasing  function  of  i  and  hence  can  cross  the  con- 
stant line  A  at  most  once. 

Let  X  and  y  be  the  number  of  arrivals  and  services, 
respectively,  in  a  given  time  period. 

Then 


E  p.  .  K(j)  =  Z  Pr  (number  of  arrivals  equals  x)  *K(i-^x) 


j=0 


ID 


=  E   K(i  +  x)  =  E   E   K(i  +  x) 
x  X  y 


and  also. 


2  q.^  K(j) 
j=o  ^^ 


Z  Z  Pr (x  arrivals) 
X  y 

K(i+x-y) 


Pr  (y  services) 
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=  E  E   K(i+x-y) 
X  y       ^ 


where 


K(n)  =  0  for  n  <  0. 
Since  K  is  nondecreasing 


D.  ^  E  E   [K(i+x)  -  K(i+x-y)]  >  0  ^-""^ 
1    X  y  ■''•'_ 

or, 


I  K(j)  [p.  .  -  q.  .]  >  0 
j  =  0        ^-'     ^^      ~ 


Now,  allowing  j  =  i  +  h,  (2)  can  be  rewritten  as 
6K(i  +  h)  +  (1  -  e)  K(i)  >_  K(ei  +  eh  +  i  -  ei) 

=  K(i  +  eh) 
or 

eK(i  +  h)  -  eK(i)  >   K(i  +  eh)  -  K(i) 


To  say  this  we  must  ensure  that  both  s  p. .K(j)  and 
Z  q. .K(j)  are  finite.   To  show  convergence  employ  the  ratio 
test  as  follows: 

.'Pij^^^^  =  .^  (j-i)!      ^(^^ 
j  =  0   -■        3  =  1   -' 


so  that 


, j-i+1  -A 
X-*     e 


Pi^j^iK(j+i)  =  ^^pinfr-    ^(^+1)  ^-^ 

, j-i+l  -\ 

p.  .K(j)  =  -j-. — ^T-T and  K(j)  and  the  limit  as  j-><=°  of 

^1,3   -^     (^-i) 


'^    \,.^ =  -7^   .,,    J;V?t    i^ust  be  <1.   This  holds  for 

a  wide  class  of  functions  such  as  exponential  and  polynomials 
and  will  be  assumed  true.   Certainly,  then,  Z  q. .K(j)  will  also 

converge  as  it  is  less  than  or  equal  to  Z  p. .K(j). 
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or  as  an  alternate  definition 

K(i  +  h)  -  K(i)  >_  ^    [K(i  +  6h)  -  K(i)  ]         (2') 
which  must  hold  for  all  0  <_  9  <_  1  and  eh  integer.   In  particular 
it  miast  hold  for  9h  =  1,  i.e., 

K(i  +  h)  -  K(i)  >_   h[K(i  +  1)  -  K(i)] 
for  all  integral  h.   As  a  special  case  when  h  =  2 

K(i  +  2)  -  K(i)  >_  2    [K(i  +  1)  -  K(i)] 
or 

K(i  +  2)  -  2  K(i  +  1)  +  K(i)  >_   0  (3) 

Denote  by  A^^'k  as  the  j^^  difference  of  K(i).   Then 

A  ^■'■^K  =  K(i  +  1)  -  K(i) 

A^^^K  =  A{A^-'-^K)  =  K(i  +  2)  -  2  K(i  +  1)  +  K(i) 
Notice,  however,  that  this  second  equation  is  just  equation  (3). 
Hence,  we  have  proven  the  following  lemma. 
Lemma  1:   The  second  difference  of  functions  with  the  convex 

property  is  non-negative. 
Therefore,  we  can  conclude  that 

A^-'-^K  =  K(i  +  1)  -  K(i)  is  nondecreasing.       (4) 

Now  consider 

A^-"-^  [K(i  +  x)  -  K(i  +  X  -  y)]  = 

K(i  +  X  +  1)  -  K(i  +  x)  -  K(i  +  X  -  y  +  1)  +  K(i  +  x  -  y) 
But  this  is  non-negative  since 

K{i  +  X  +  1)  -  K(i  +  x)  >_  K(i  +  x  +  1  -  y)  -  K(i  +  x  -  y) 
from  the  property  in  (4).   Hence, 
K(i  +  x)  -  K(i  +  X  -  y) 


16 

is  nondecreasing  in  i  for  all  x  and  y.   Then  surely, 

D.  =  E   E   [K(i  +  x)  -  K(i  +  x  -  y) ] 

must  be  nondecreasing  in  i  and  hence  crosses  A  at  most  once. 

This  proves  the  following  important  theorem. 

Theorem  1:   For  n=l,<t>  (i,  ct )  has  a  solution  given  by  a  control 
limit  rule,  i.e.,  there  exists  a  unique  number  such  that 
if  and  only  if  with  one  period  remaining  there  are  more 
than  this  number  in  the  system,  will  a  closed  server  be 
opened. 
For  the  case  when  the  server  is  open  with  one  period  re- 


maining. 


(j>,  (i,  a)  =  min 


K(i)  +  B  +  a  E   p.  .  K(j) 


j  =  0 


ID 


K(i)  +  C  +  a  Z    q  .  K(j) 


j  =  0 


ID 


This  leads  to  observing  when 

K(i)  +  B  +  a  E  p^  .  K(j)  (j)    K(i)  +  C  +  a  Z  q^^  K(j) 
or, 

"l   [Pij  -  qij]  K(j)  (f)  ^=  3 

The  preceding  theorem  then  also  applies  here,  hence   guaran- 
teeing a  unique  point  at  which  the  server  would  be  closed. 


Proof  for  arbitrary  n 

For  n>l  it  must  be  shown  that  (j*,  i*)  policies  are  opti- 
mal for, 
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^    (i,  a)  -  min 
n 


K(i)  +  a  I    p    <^     (j,  a) 
j  =  0   ^ 


K(i)  +  A  +  C  +  aE   q..  (j)^_,  (j,  a) 


j  =  0 


ij   n-1 


and 


<l>J(i,  a)  =  min  [  K(i)  +  B  +  a  E  p^   <t)°_^  (j,  a) 


K(i)  +  C  +  a  E  p    <(>';    (j,  a) 

xj      n-i        ; 

Before  this  is  shown,  let  us  prove  the  following  lemma 

Lemma  2:   For  n=l,  i*  ^  j* 

Proof:   For  n=l,  i*  must  satisfy 


K(i*)  +  a  E  p^^   K(j)  1  K(i*)  +  A  +  C  +  a  E  q^^   K(j)  or, 


E  [p..  .  -  q...]  K(j)  >  ^±^ 
Similarly  j*  must  satisfy,  K(j*)  +  B  +  aEp.^.  K(j) 


<.  K(j*)  +  C  +  E  q     K(j)  or, 

Z  [p.  .  .  -  q.  .  .]  K{j)  <  ^^ 

Since  A  +  C  >  C  -  B  and  E  [p . .  -  q. .]  K(j)  is  nondecreas- 
ing,  i*  >_   j*  must  hold. 

Now  let  us  establish  the  existence  of  the  values  (j*,  i*) 
for  n=2.  Vie   have  that 


^2^^'    °')  -   ^i^    ^(^^  "•"  c'  ^  P^-j  "^i    (j'  ") 


K(i)  +  A  +  C  +  a  E  q^  .  ^       (j,  a) 


>   (5) 


and 


18 


(^2  (if  a)  =  min 


0,. 


K(i)  +    a  I  p^  .  (jj-^Cj/a)  + 


B 


K(i)  +  C  +  a  Z  q^  .  (J)j^(j,a) 
Just  as  before  we  can  write 


(6) 


I  P^j  •{'^(jfa) 


^x  ^v  *l(i+^'"^ 


and 


(j)^(i+x-y,a) 


2  q^j  ^^(j/a)  =  E^  E^  (t,^(i+x-y,a)  , 

and,  as  before,  if  it  can  be  shown  that  (^,   (i+x,a) 
is  nondecreasing,  then  so  is 

J:  [P^j  (t)3_(j/a)  -  q^j  <j)-L(j,a)  ] 

which  implies  that  the  alternatives  in  (5)  and  (6)  can  cross 
at  most  once  and  the  result  will  be  established. 


(|),  (i+x,a)  =  min 


-  min 


K(i+x)  +  Z  a   Pi+x,j  ^^^^ 


K(i+x)  +  A  +  C  +  a  Z  q.  ,^  •  K(j) 

XtX  ,  J 


K(i+x-y)  +  B  +  aZp^^j^_y^j  K(j) 


K(i+x-y)  +  C  +  aZq^^^_y^^  K(j) 


(I) 

(II) 

(III) 
>   =  *  j^(i+x-y,a) 
(IV) 


where  the  (I) ,  (II) ,  (III)  and  (IV)  notationally  correspond 
to  that  particular  alternative. 

It  is  necessary  to  consider  only  the  possibilities  when 
the  minima   are  (I)  and  (III),  (I)  and  (IV),  and  (II)  and  (IV), 
i.e.  (II)  and  (III)  cannot  concurrently  be  minima  by  lemma  2. 
Now,  let  us  formally  prove  the  following. 
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Theorem  2:   (t>^(i+x,ct)  -  (i^(i+x-y,a)  is  nondecreasing. 

Proof:   When  (I)  and  (III)  are  minima  41,  (i+x,a)  -  41 1"  (i+x-y,a )  = 

K(i+x)  -  K(i+x-y)  +   aZ[p.^^^.  -  Pi+^.^^j]  K(j)  -  B 

which  is  nondecreasing,  the  first  two  terms  from  assump- 
tion (i)  and  the  second  two  since  K(i+1)  -  K(i)  is  nonde- 
creasing. 

When  (I)  and  (IV)  are  minima, 

(})j^(i+x,a)  -  (^  j^(i+x-y,a)  =  K(i+x)  -  K(i+x-y) 

+  aZ[p.     .-q.,     ■]K(j)-C 
which  is  nondecreasing.   This  can  be  shown  be  replacing 
the  bracket  by 

Ip-    .-p.      .+D.      .-Q.      .]K(i) 
'^i+x,3    ^i+x-y,:    "i+x-y,:    ^i+x-y,:^  ^-" 

which  is  the  sum   of  two  nondecreasing  functions. 

Finally  when  (II)  and  (IV)  are  minima 

<)>J(i+x)  -  (|>j^(i+x-y,  j)  =  K(i+x)  -  K(i+x-y) 

+  a  Z  [q.^   .  -  q.^     .]  K(j)  +  A 

which  is  nondecreasing  as  was  the  first  case  considered. 

This  completes  the  proof. 

This,  then, leads  to  the  following: 

Corollary  1:   There  exist   numbers  (j*,  i*)  for  n=2  such  that 

(J)- (if a)  and  <^^{i.,Qi)    have  optimal  solutions  of  the  required 

form. 

Now,  let  us  introduce  the  hypothesis  that, 

for  n=3,  4,  ...  ,  k,<))  (i,a)  and  (J)  (i,a)  have  control  limit 

n  n 
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rules  and,  also, 

^  [Pij  *n-l(^'")  -  ^ij  *i-l^^'")^ 
is  nondecreasing. 
It  becomes  necessary  to  show  next  that 


'''k+l^^'"^  =  min 


and 


<J>j^^3^(i/a)  =  min 


K(i)  +  a  E  p^j  *°(j,a) 


K(i)  +  a  I  q^   ^^{j,^)  +  A  +  C 


K(i)  +  B  +  ct  Z  p^^  *°(j,a) 
K(i)  +  C  +  a  I  q^   *J(j,a) 


have  the  required  property.   This  is  done  in  precisely  the 
same  manner  as  before,  i.e.,  it  is  shown  that 

is  nondecreasing  by  looking  at 

(}ij^(i+x,a)  -  (^j^(i+x-y,a) 
Again,  we  shall  first  show 
Lemma  3:   For  each  n,  i*  ^  j*. 
Proof:   At  i*,  the  following  must  be  true, 

K(i*)  +  a  I   Pi*j*°_i(J'«)  L   K(i*)  +  A  +  C  +  a  Z  qi* j*n-l ^^ ' "^ 

or 

^i*j  n-1  -*      ^1*:)  n-1  -■     —  a 
At  j*  the  following  must  hold, 
K(j*)  +  B  +  a  E  Pj*j*°_i(J/'=')  lK(j*)+C  +  aE  P  j  *  j  *^-l  ^^  ' ''^ 


or 


21 


C-B 


'   IPj^j  *n-l(^'^)  -  ^3.*.  ^'-lO.c')]  1^ 


3^j      n-i—  ■    -3'^:] 
Since  A  +  C  >  C  -  B  and  the  left  sides  are  nondecreasing 
by  the  induction  assumption/  it  naturally  follows  that 
i*  >^  j*. 

Theorem  3: 


(|)j^(i+x,a)  =  min 


< 


K(i+x)  +  a  E  p 
K(i+x)  +  a  Z  q 


i+x,j  *k-l^=''='^ 

i+x,j  *k-l(3,-)  +A  +  C 


-min  K(i+x-y)  +  B  +  « ^  Pi+^-y  *k-l^^'"^ 


(III) 
/  =  ())j^(i+x-y,  j) 


(IV) 


K(i+x~y)  +  C  +  a  Z  q.^     .  (j)f  ,  (j  ,a) 
-^  ^i+x-y,3   K-1  -^ 

(where  the  (I),  (II),  (III)  and  (IV)  are  unambiguous)  is 
nondecreasing. 
Proof:   From  lemma  3  we  see  once  again  that  it  is  only  neces- 
sary to  consider  when  (I)  and  (III),  (I)  and  (IV),  and 
(II)  and  (IV)  are  minima. 

When  (I)  and  (III)  are  minima  the  difference  is 
K(i+x)  -  K(i+x-y)  +  a  Z  [p .  ,   .  -  p.,      .]  A,  ,(j,a)  -  B, 

which  is  nondecreasing;  the  summation  is  nondecreasing 
which  follows  from  the  discussion  on  piecewise  convex 
functions  in  Appendix  I. 

When  the  minima  are  (I)  and  (IV)  we  have 
K(i+x)  -  K(i+x-y) 

■^  °^  IPi+x,j  *k-l(^'^)  -  ^i+x-y,j  *J-1^J'«^^  -  ^- 
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This  is  nondecreasing  since 

Pi+x,j  *k-l(=J'")  -  ^i+x-y,j  *k-l^^'"^ 

^Pi+x,j  ~  Pi+x-y,j]  ^^-l^^'"^ 

+  Pi+x-y,j  *k-l(3,a)  -  qi+x_y,j  ^J-l  ^^'^^ 
and  the  first  two  terms  are  nondecreasing  by  the  first 
part  of  this  proof  and  the  second  two  by  the  induction 
hypothesis. 

Finally,  when  (II)  and  (IV)  are  minima 
K(i+x)  -  K(i+x-y)  +  aZ  [<i^^^,^'<:i^^^_y  ,^U^_^ij ,-)    +  A 

is  nondecreasing  by  again  using  the  results  of  the  dis- 
cussion from  piecewise  convex  functions.   This  completes 
the  proof. 
Hence,  it  is  established  that  ^i^+i^J'^')  ^^^  ^k+i^J/") 

have  control  limit  optimal  rules. 
Thus,  we  have  established. 

Corollary  2:   For  any  n,  (j)  (i,a)  and  ((>  (i,a)  have  solutions 
which  are  control  limit  rules. 


Extension  to  Other  Objectives 

The  theory  behind  the  next  result  is  due  largely  to  Bellman. 
Theorem  3:   A  control  limit  rule  is  optimal  for  the  infinite 

horizon  model. 
Proof:   Let  us  prove  this  theorem  for  (fi  (i,a);  the  other  follows 

similarly. 


2 
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We   know   that 


♦  j^Ci/O  )    =   min        K(i)    +aEp..    ^    _       (j,a) 


K(i)    +A   +   C+aZq..    (})-'-       fj, a) 

^n      n-l   -" 


Define, 


"^l^^n-l^    =   ^^^^    "^  "^  Pij   *n-l^^'"^ 


and 


'^2^'''n-l^    =    ^^^^    +   A   +    C    +   cxZ  q^^    ^J-l^^/") 
Then, 

-i-^Ci,")    =  min        [T.     (*°   ,)] 
"■  j  =  l,2  ^         ^"-^ 

Let  j  (n)  =  index  which  gives  the  min  [T .  (4>  )] 

j=l,2   ^   ^ 


which  implies  that 


(*n-l)  IT 


j  (n-l)  ^n-1  —   j  (n)  ^n-1 


u?_.) 


(^°(i,o)  =  T. 
and 

Subtracting, 

l*n^i'^^  -  *n+l(i'«^ll^^^  fl^j(n-l)^*n-l^  "  ^j  (n-l)  ^^n^  I  ' 

l^j(n)(0  -^j(n)(*n-lM3 

<  max[lT  (^°   )-T.  (^°)  |] 
~j  =  l,2   J   ^  -^    J   " 

<_  max[  |K(i)+a  Z  Pin^n-l^^"^^ 


-  K(i)-a  Z  F^.*j^(j,a)  |  ,  |K(i) 
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A+C+a  I   q^^J_-^(j,a)-K(i)-A-C-a  I  q^j<^J(j,a)  ]  ] 

<_   max[  I  a  I  Pj_^<>^_i  (jfa)-alp^.<i)j^(j,a)|. 


1  a  E  p^.  l<{'j^_-,_(j,ct)-(|>^(j,a) 


Hence 


max  I  4)^(i,a)-(|)^_^^(i,a)  \  <a.  Z  p^  .  max  |  4i^_-]_  (i  ,  a  ) -(^^  (i  ,  a)  ] 


<_a  max|  4)^_^  (i,a)-({)^(i,a)  I 


1 


Define 


i 
and  hence 

2  n 

U  ,,<au  <a  u   ,<  ...<a   u, 
n+1—  n—   n-1-    —    1 

and  therefore 

CD  00  U 

Z  U  ^  ,  <  Z   a"^  u,  =  T 

A  n+1—   -      1    1-ot 
n=0     n=0 

and  we  conclude  that 

oo 

n=0 

converges  uniformly  and  that  <^     (i/a)  converges  to  <j)  (i/a) 
for  all  i,  where  <j)  (i,a)  satisfies, 


(j>  (i,a)  =  min 


K(i)  +  a  Z  p^.*  (j,cc) 

K(i)  +  A  +  C  +  a  Z  q^.*''"(j,a) 


completing  the  proof, 
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Now  the  significance  of  restricting  our  decision  set  to 
be  finite  when  the  system  is  closed  will  become  apparent.   We 
shall  assume  that  for  any  e>0,  3an  M(e)3:  Pr  [X  >  Hit)]    <e 
and, henceforth,  consider  opening  only  in  a  finite  number  of 
states.   When  the  server  is  open,  assuming  that  the  arrival 
rate  is  less  than  the  service  rate  is  sufficient  to  guarantee 
that  the  queue  size  will  remain  finite;  hence,  only  a  finite 
number  of  decision  points  can  occur. 

Finally,  we  can  prove  the  following: 
Theorem  4:   A  control  limit  rule  is  optimal  for  the  averaging 

cost  criterion. 
Proof:   Let  i   be  the  optimal  crossing  point  of  4)  (i,a).   Let 

{a_},  with  lim  a^  =  1,  be  a  sequence  such  that  i    =  i* 

for  all  s.   Since  there  are  at  most  a  finite  nurrier  of 
States  where  the  system  will  be  opened,  such  a  sequence 
and  i*  must  exist.   Let  R  be  any  rule  which  is  not  con- 
trol limit.   Then  certainly 


Hence, 

lim  (l-Qg)   ^^(i'as)!   lini  (1-a  )*  ^^'^s^ 

and  from  Arrow,  Karlin  and  Scarf, 

(^^   =  lim  (l-Cg)  ^j^^i'C'g)  where 
s->-=° 

0       1   T 
1^^   =  lim  —  Z     g(X.  )  and  where 
R   T-oo  T  t^i     t 

g(X  )  denotes  the  cost  incurred  when  observed  in  state 
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X.  .   Therefore,  fji^  i.  '{>   where  <}>   is  the  optimal  value 

t  K 

under  the  control  rule  i*  and  hence  i*  is  the  optimal 

rule.   The  proof  is  identical  for  <})  . 


Closing  When  There  is  a  Non-empty  Queue 

Many  have  conjectured  that  a  server  will  only  be  closed 
if  no  customers  are  awaiting  service.   Others  have  said  that 
the  closing  point  is  some  function  of  the  system  parameters. 
Let  us  answer  this  question  by  showning  under  what  conditions 
the  following  theorem  is  true. 
Theorem  5:   The  server  will  never  be  closed  when  any  customers 

are  present  in  the  system. 

Let  us  assume  that  we  have  established  a  unique  point, 
say  i*,  at  v;hich  the  server  should  be  turned  on  or  opened. 
Let  us  also  assurrie,  contrary  to  the  theorem,  that  the  optimal 
operating  policy, n,  designates  to  close  the  server  at  some 
number  j*,  0  <  j*  <  i*. 

Since  the  process  is  in  equilibrium,  we  know  that  with 
probability  one,  state  j*  (or  less)  will  be  reached  once  the 

server  has  been  turned  on.   Hence,  the  process  will  alternate 

[21 
betv/een  states  j*  and  i*. 

There  exists,  however,  a  number  k,  0  <  k  <  j*,  such  that 
the  expected  cost  under  policy  n  is  equivalent  to  the  expected 
cost  under  policy  n, ,  where, 

n, :   Open  the  server  when  there  are  i*  -  j*  customers  in 


12] 

In  fact  it  will  alternate  between  states  less  than  or 

equal. to  j*  and  greater  than  or  equal  to  i*. 
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line,  close  when  the  line  is  empty  and  keep  k    customers 

on  the  side  at  all  times. 
Define  now  policy  II   as  follows. 

11-:   Open  the  server  when  there  are  i*  -  j*  customers 

in  line,  close  when  line  is  empty,  and  after  some  finite 

time  serve  the  k  customers. 

Now  for  each  possible  objective  function  consider  the 
following  three  policies. 

1)  Total  Expected  Undiscounted  Cost--Here  the  additional 
expected  cost  under  n  ,  that  of  holding  the  k  customers  to 
the  side  becomes  infinite,  while  the  additional  expected  cost, 
that  of  serving  the  k  customers,  is  certainly  finite  and  hence 
II2  dominates  11   and,  therefore,  also  n. 

2)  Average  Cost — The  holding  of  the  k  customers  under 
policy  n  occurs  for  each  period  and  hence  adds  a  positive 
quantity  to  the  average  period  cost.  However,  the  cost  of 
serving  the  k  customers  occurs  for  only  a  finite  number  of 
periods  and  hence  adds  nothing  to  the  average  period  cost. 
Hence,  again  n   dominates  both  n  and  II  . 

3)  Total  Expected  Discounted  Cost--Here,  both  the  holding 
cost  of  the  k  customers  along  with  the  cost  of  serving  these 

k  customers  is  finite  and  hence  a  comparison  is  necessary. 

For  a  discount  factor  a  the  cost  of  holding  the  k  customers  is 

at  least  ~       t^  /i  \ 

K(k)    I   a^  =  ^(^) 


n=0 


13] 

If  k  is  not  integer  keep  instead  [kj  customers  off  to  the 

side,  so  if  anything  n   dominates  n. 
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The  cost  of  serving  the  k  customers,  which  we  assume  takes 
a   periods,  along  with  the  holding  cost  of  these  customers, is 
bounded  above  by 

£-1        i-1 
K(k)   I  a^  +  C   la 
n=0        n=0 

=  [K(k)  +  C]  I   _    I 
To  choose  n   over  n   we  must  require  that 

1  -  a^  <  K(k) 


[K(k)  +  C]  ^-r^   <  Y-=^ 

"   ^  K(k)+C 
Hence,  if  the  discount  factor  is  large  (a  reasonable  assump- 
tion), n-  dominates  n  as  well  as  n^. 

4)  Finite  Horizon-In  this  case  it  is  possible  to  close 
the  server  with  customers  present.  This  event  occurs  if  the 
holding  cost  is  small  compared  to  the  operating  cost. 

Again,  the  difference  between  analyzing  my  system  and 
that  of  analyzing  the  system  after  each  arrival  and  departure 
is  that  k  in  the  preceding   argument  is  just  j*. 

Concluding  Rem.arks 

Finally  we  can  say  for  the  single  server  queue  that  an 
optimal  stationary  policy  always  has  one  of  the  following  forms 

a)   Always  keep  the  server  closed  until  M  is  reached. 
This  case  arises  when  discoxinting  is  used  and  the  holding  cost 
is  small  compared  to  the  operating  cost. 
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b)  Never  close  the  server.  -  This  may  occur  v/hen  the 
server  set-up  and  shut-down  costs  are  large  compared  to  the 
operating  cost. 

c)  There  exists  a  pair  (j*,  i*)  with  the  interpretation, 
open  the  server  only  if  there  are  at  least  i*  customers  present; 
close  the  server  when  there  are  no  more  than  j*  customers 
present.   The  important  special  case  of  j*  =  0  is  discussed 

in  Theorem  5  and  that  which  followed. 


CHAPTER  III 


COMPUTATIONS  FOR  SINGLE  SERVER 


Once  existence  of  optimal  solutions  has  been  established 
it  becomes  necessary  to  find  an  i*  which  might  make  it  worth- 
while to  "dismantle"  the  service  station.   The  closing  of  the 
service  station  under  a  (0,  i*)  doctrine  will  cause  increased 
holding,  and   set-up  and  shut-down  costs.   The  savings  from 
the  partial  elimination  of  the  station,  during  an  idle  period, 
might  not  outweigh  these  increased  costs.   Hence,  an  optimal 
i*  must  be  found  and  the  cost  therein  achieved  must  be  com- 
pared to  that  of  a  simple  queue,  i.e.,  never  close  the  server. 
A  simple  computational  scheme  is  developed  to  make  this  com- 
parison. 

The  second  part  of  this  chapter  makes  use  of  Howard's 
approach  in  analyzing  the  system  from  a  computational  view- 
point.  Description  of  the  algorithm,  calculation  of  probabil- 
ities,along  with  some  interesting  results  are  presented. 

Averaging  Computational  Scheme 
Calculation  of  Average  Queue  Length 

Let  us  consider  an  averaging  cost  criterion,  i.e.,  find- 
ing the  optimal  policy  for  the  average  cost  per  cycle.   By  a 
cycle  we  mean  the  following: 
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Let  us  begin  at  the  moraent  our  ser^/ice  station  has  been 
closed  dov/n  -  there  are  no  customers  waiting  for  service. 
Call  phase  I  that  time  it  takes  for  i  customers  to  accumulate 
before  the  service  station.   By  the  second  phase,  is  meant 
the  time  from  when  the  i    arrival  enters  the  system  until 
the  end  of  the  present  time  period.   Let  the  average  number 
of  arrivals  in  this  phase  be  denoted  by  a ' .  Let  the  third 
phase  consist  of  servicing  these  i  +  a ' customers  along  with 
their  generation  arrivals  (all  arrivals  occurring  while  the 
original  i  +  a' are  being  served)  until  there  is  no  queue  and 
our  service  station  is  once  again  closed  dov/n.   Phases  I,  II 
and  III  make  up  one  cycle. 

, The  average  time  for  phase  I  is  i  /  X,  the  assembly  time 
for  i  customers. 

Let  F (t)  be  the  distribution  of  the  length  of  time  for 
phase  II.   Let  E(t)  be  the  expected  length  of  time  for  phase  II. 
Then, 


00     CO  ,      .  T  00 

•=  /   y  ^o-^t  (Xt)^   .^,^,  ^     J 


J-  j^-Xt  jxti        ^^^^^    ^    J  ^^  ^p^^^  ^  ^^^^^ 


0  j  =  0       -'•  0 


and  hence  E(t)  =  f-  " 

A 


Let, 


p  =  1  <1) 


be  the  traffic  intensity.   In  addition  define  P,  and  P   as 

-^  be 

the  time  of  busy  and  clear  periods  for  ordinary  queuing  sys- 
tems.  Then,  because  of  the  Poisson  arrival  process, 

P^  =  1/A  (2) 


32 

Also,  the  fraction  of  time  the  server  is  busy  is  given  by 

^b 
P  = 


b   c  (3) 

On  combj.ning  (1),  (2)  and  (3),  we  obtain, 

P   =  -^ 

b    y-A 

This  is  the  expected  length  of  the  busy  period  for  one  cus- 
tomer and,  hence,  the  expected  duration  of  the  third  phase 
is  (i+a')/(y-A) . 

The  average  length  of  a  cycle  can  then  be  easily  obtained 
by  summing, 

i    a'  .  (i+g')  ^  i(y-X)+a'  (M-A)  +  (i+a'  )X 
A    A   "^    y-A  A  (y-A) 

y  (i+g' )   ^  i  +  g' 

A(y-A)      A(l-p)  (4) 

Also,  the  fraction  of  time  the  system  is  in  each  of  the  phases 

iS/ 

i_  =  i(i-p ) 

XT    i  +  a"^  (5) 

3l  =  g'  (1-p) 

AT    i  +  a'  (6) 

i  +  g'  _  cj^ 

for  phases  I,  II,  and  III  respectively. 

Let  us  adopt  the  following  notation: 

p,  .  =  Pr   There  are  k  customers  in  the  system  and  there 
are  j  available  servers 

K  =  0 ,  1 ,  ...  and  j  =  0 ,  1 

th 

t. -A  random  variable  denoting  the  length  of  time  from,  the  i 


arrival  to  the  and  of  the  next  observation  point. 

T   -  A  random  variable  denoting  the  time  between  the 
s 

arrival  of  a  customer  in  the  third  phase  until  the  com- 
pletion of  the  particular  service  in  progress. 
T   -  A  random  variable  denoting  the  time  betv;een  the 
arrival  of  a  customer  in  the  second  phase  until  the 
end  of  the  second  phase. 
W  -  Average  queuing  time 
L  -  Average  queue  length 

Let  us  note  some  important  relationships: 
Since  phase  I  is  just  the  arrival  of  customers  through 

a  Poisson  stream  all  states  in  phase  I  will  occur  with  the 

same  probability.   Hence,  from  (5) 

PkO  =  tS-.      K=  0,  1 i-1  (8) 

Also,  from  (6)  the  probability  of  being  in  phase  II  is 

Finally,  the  associated  probability  for  phase  III  is, 

"   p„,  =  P  (10) 

K=l   ^^ 

4 

Next,  Cox  and  Smith    show  that  the  forward  delay,  x,  is 

distributed  so  that 

2 

»  . 

T"'  (11) 

where  t  is  the  original  random  variable  and  v,  the  coeffi- 
cient of  variation  of  t.   For  the  exponential  distribution, 
V  =  1. 


E(T)  =  E(t)  i^''^''  ) 
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Now,  we  proceed  to  analyze  the  queuing  time  of  customers 
arriving  in  different  phases.   For  a  customer  who  arrives  in 
phase  I,  where  there  are  K  customers  present,  his  waiting 
time  is, 

(i-l-K)/A4-  E(t^)  +  (K+1)/m 
An  arrival  in  the  second  phase  when  the  system,  is  in  state  K 
can  expect  to  v/ait, 

E(t^)  +  (K+l)/p 
Finally,  a  customer  in  the  third  phase,  when  K  are  present 
at  his  arrival,  will  wait  on  the  average, 

E(t  )  +  K/u 

Now  we  are  in  the  position  to  write  down  the  average 

queuing  time  W: 

i-1 
W  =   .E   p_,-[(i-l-K)/A  +  E(t,)  +  (K+l)/y)] 
K=0    ^^  ^ 


+   E  p    [E(x  )  +  (K+l)/y] 
K=i   ^°      ^ 


+   E  p     [E(t  )  +  K/y] 
K=l   ■^-'       ^ 

Insertion  of  (1),  (8),  (9),  (10),  and  (11)  yields, 

i+a'         i+a '        2 

19 

From  Little's     formula, 

L  =  XW 
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the  average  queue  length  is  then, 

2.             .  ,  .  T  .       ,         i  +  v 
L  =  p  +  P_  +  liiZll  +  _^  [i+^,  ( !_)] 

1-p    2  (i+aO    i+a'         2 

which  is  the  average  queue  length  of  a  simple  queue,  given 
by  the  first  two  terms  (Pollczek-Khintchine) ,  plus  the  addi- 
tional queue  length  due  to  the  intermittent  operation  of  the 
server. 


Optimi:^ation 

Now  it  is  possible  to  write  down  the  cost  function  for 

the  averaging  criterion. 

A+B 
First,  in  one  cycle  a  cost  — —  v;ill  be  incurred  due  to 

set-up  and  shut-down  of  the  server.   Also  we  can  expect  a  con- 
tribution of  K(L)  as  the  average    holding  cost  per  cycle. 
Finally,  in  one  cycle  there  is  savings  due  to  the  decreased 
use  of  the  server.   The  server  is  being  used  only  when  in 
phase  III  and,  hence,  the  cost  incurred  from  its  use  is  expected 
to  be  pC/T. 

Therefore,  the  average  cycle  cost  is 

r  (i)  =  K(L)  +  AJlZPl(A±Bi  +  pC/T 
X.  1+a'  ' 

To  find  the  optimal  opening  point,  i,  we  may  differenti- 
ate C  with  respect  to  i,  set  equal  to  zero,  and  solve  for  i 
(an  integral  value  is  naturally  required) .   However,  this  is 
impossible  as  given,  as  it  is  necessary  for  the  form  of  K(-) 
to  be  explicitly  given. 
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A  Specific  Case 

First,  for  the  sake  of  example  only,  and  second,  to  com- 
pare the  results  with  some  previously  cited  works,  let  us 
assume  that  the  holding  costs,  K(-)/  are  linear  with  respect 
to  the  averaging  values.   Then, 

,  (A^BUd-p)  ,  ^,^, 
Differentiating , 

8^    K.  (i+a')  (i-^)-  ^  (i-1) 


3i  (I+c0^2 


or, 


(i+a-)^  [i+d)^ 


Ki^  +  2Ka!L  +  Ka'[a'-l-a?v^^]  -  2(A+B)X(l-p)  =  0 


=  0 


or. 


i2    +    2c/i    -f    [a'(a'-l-a'v    2)    _    2(A+B)X(1-p)]    ^    ^ 
which  has   roots, 

(      -2a'^V4a'^-    4     [a'(a--l-a--o    ^)    -       2(A+B)A(l-p)T 

t  K 

or, 

-a-   ^     V--'    -     [c^(c^-l-a'v    2)     .    2(A^B)A(l-p)] 

t  K 


)/2 


Simplifying  we  obtain. 


-  ""V 
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2(l-p)A(A+B)  ^  ,,(1,.,   2 J 


K         -x^.^.^ 
The  positive  root  yields  the  minimum  and,  hence 

i  =  -a-  +  y2(l-p)A(A+B)^  a'(l  +  V)  (12) 

The  optimal  solution  is  then  obtained  by  comparing  the 
cost  of  the  nearest  positive  integers  [i^]  ,  [i^+1]  and  the  pos- 
sibility of  permanent  server  availability.   This  latter  cost 
is  just 

2 

Cg  =  K.  [p+  ^]  +  C/T 

That  is,  i*  is  defined  by 

C(i*)  =  min  [C[i],  C([i+1]),  C^] 

(in  the  last  case  i*  =  0). 

It  is  interesting  to  note  that  the  value  of  _i  obtained 
by  differentiation  is  not  a  function  of  the  running  cost,  C, 
of  the  server.   Actually,  this  result  is  not  surprising  as 
over  a  reasonable  length  of  time  the  same  number  of  customers 
will  be  served  on  the  average;  hence,  the  server  will  be  running 
for  the  same  length  of  time. 

Finally,  if  the  system  is  observed  after  each  departure, 
rather  than  at  equal  intervals,  phase  II  disappears.   Therefore, 

(12)  becomes 


i=V 


(A+B)A (1-p) 


K 

This  result  agrees  precisely  with  that  of  Heyman's, 
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Computational  Experience  Using  Howard's  Algorithm 

Brief  Description  of  Algorithm 

Given  a  process  which  recieves  a  reward  q^  at  the  begin- 
ning of  a  period  when  found  in  state  i,  then,  if  V^ (n)  is 
the  present  value  of  the  total  expected  return  when  in  state 
i  with  n  transitions  remaining,  we  obtain 

M 

V^(n)  =  q^  +   aZ   [p^^  Vj(n-l)] 

for  each  i  and  n  where  a  is  some  discount  factor  and  a  total 

of  M  states. 

9 
With  the  introduction  of  alternatives,  Howard  has  devel- 
oped an  algorithm  which  minimizes  these  present  values  for 

an  infinite  horizon  problem,  i.e.,  replacing  V^  =  lim  V^ (n) 

n^°° 
for  V.  (n)  .   This  routine  can  be  summarized  by  the  follov/ing 

flowchart  in  Figure  2. 

The  iteration  process  can  be  entered  in  either  box  and 
Howard  guarantees  that  each  cycle  will  yield  a  present  value 
at  least  as  good  as  the  one  previously  obtained.   He  also 
guarantees  that  there  is  convergence  to  an  optimal  policy 
and  this  is  noted  by  having  policies  on  two  successive  itera- 
tions identical. 

In  the  problem  at  hand  either  of  two  decisions  may  be 
made  in  any  of  the  2lA   states.   The  states  are  characterized 


•^  -'   Veinott    has  guaranteed  convergence  in  a  finite  number  of 
steps. 
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Value  Determination  Operation 

Use  the  p. .  and  the  q.  for  the  given  policy  to  solve 

J  the  set  of  equations 
~^ 

N 
V.  =  Iq.  +  a Z   p..  V  ]    i  =  1,2,...,N 
^  j=l   ^^  ^ 

for  all  present  values  V. . 

1 


Policy  Improveiaent  Routine 

For  each  state  i,  find  the  alternative  k*  that  mini- 
mizes 

^^i  -^  ".^^  Pij  ^j^  Iv 

using  the  present  values  V.  from  the  previous  policy, 

Then  k*  becomes  the  new  decision  in  the  i    state, 

k*  k* 

p. .  becomes  p. .  and  q-   becomes  q. . 


Figure  2:   Policy  Interation  with  Discounting 
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by  the  nuniber  in  the  system  0,  1,  ...,  M-1  and  whether  the 
server  is  closed  or  opened.   The  cycle  was  entered  in  the 
policy  improvement  routine  by  choosing  all  the  present  values 
as  zero.   The  first  step,  therefore,  was  minimizing  immediate 
returns,  i.e.,  choose  the  alternative  that  chooses 

K(i) 
q^  =  min   < 

K(i)  +  A  +  C 

when  the  server  is  closed  and 


K(i)  +  B 


q.  =  mm 


> 


K(i)  +  C 

when  the  server  is  opened.  The  new  present  values  were  then 
calculated  and  iteration  process  continued  until  convergence 
was  obtained. 


Calculation  of  Probabilities 

In  order  to  make  use  of  the  algorithm  explicit  values 
for  the  transition  probabilities, p . .  and  q. .  had  to  be  known. 

When  the  server  was  closed,  Pj^^  /  the  probability  of  going 
from  i  to  j  in  one  time  period, was  simply  the  probability  of 
j-i  arrivals,  where  the  arrivals  are  Poisson  distributed  with 
parameter  X . 

With  the  server  opened,  the  probabilities  q. .  are  derived 
as  follows.   Prabhu    finds  the  Pr  (there  are  <_j  in  the  system 


at  time  t  I  there  are  i  in  the  system  at  time  0) 
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=  Pr(Q(t)  1  j  I  Q(0)  =  i) 
-  (X  +  u)  t.  a  b,  a+b  ^- (X+y  )  t,  a+j+1  b-j--l^a+b 

A(j-i)       ^'^'  A(-j-i-2)         ^•^• 

where  A(j)  denotes  the  set  of  integers  a,  b  such  that  a>0, 
b>  0  and  a-b<_j  . 

Subtracting  from  this  quantity 

Pr  (Q(t)  1  j-1  I  Q(0)  =  i) 
we  get 

Pr  (Q(t)  =  j  I  Q(0)  =  i) 
Next,  setting  t=l,  i.e.,  one  time  period,  we  establish 

^ii         a-ij-i  -"^'   A(-"  i-1)  -'-^^     A(-A-2)     ^"^^ 
Since  we  always  make  the  decision  to  open  at  state  M  and 
above,  these  probabilities  are  truncated  at  M.   This  is  neces- 
sary as  Howard's  routine  allows  for  only  a  finite  number  of 
states. 

Results 

A  FORTRAN  program  was  used  to  calculate  these  probabilities 
and  then  incorporated  into  another  which  was  Hov;ard's  policy 
improvement  routine.   Some  of  the  results  obtained  v;ere  both 
reassuring  and  enlightening.   Since,  however,  sensitivity  on 
one  parameter  was  accomplished  while  the  other  parameters  ware 
fixed  at  particular  levels  any  definite  conclusions  were  dif- 
ficult to  reach. 

Sensitivity  to  discount  factor 

Theorem  5  in  Chapter  ii  stated  that  it  was  possible  to 
close  the  server  with  customers  present  for  the  infinite 


TABLE  1 


SENSITIVITY  OF  DISCOUNT 
FACTOR  TO  HOLDING  COST 
FOR  A=1.0,  u=2.0, 
A=10,  B=10  AND  C=40 


j*   for   K(X)    = 

a 

X 

X2    +    X 

e^ 

0.9 

0 

0 

0 

0.8 

0 

0 

0 

0.7 

M 

0 

0 

0.6 

M 

0 

0 

0.5 

M 

2 

0 

0.4 

M 

4 

1 

0.3 

M 

M 

2 

0.2 

M 

M 

3 

0.1 

M 

M 

4 
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horizon  discounted  case.   For  fixed  costs,  however,  this  was 
only  true  for  small  a.   Table  1  gives  j*,  the  closing  point 
as  a  function  of  three  different  pararaeter  values.   Notice 
that  the  values  are  not  only  small  but  also  as  the  holding 
costs  became  "more  convex"  the  value  of  the  discount  factor 
when  the  server  is  closed  with  customers  present  decreases. 

Other  results 

Certainly  extreme  cases  led  to  extreme  results  as  far 
as  determination  of  i*  and  j*.   For  example,  for  high  set-up 
and  shut-down  costs  and  low  operating  costs  as  soon  as  we  open 
we  always  remain  open.   Also,  if  the  holding  cost  is  small 
compared  to  the  fixed  costs  once  we  close,  we  will  always 
remain  closed  (since  the  fixed  costs  are  constants,  this  only 
occurred  in  our  investigation  when  the  holding  cost  was  linear) - 

An  interesting  observation  was  that  the  sensitivity  of  i* 
to  the  ratio  of  the  arrival  and  service  rates  (A/y)  was  very 
slight.   However,  i*  was  quite  sensitive  to  the  magnitude  of 
X  and  y.   When  X    and  y  were  chosen  large  (>_1)  i*  turned  out 
to  be  only  0  or  1.   However,  when  their  magnitudes  were  decreased 
(X=0.1,  y=0.2)  i*  had  the  value  4.   Certainly  there  is  varia- 
tion here  depending  on  costs  but  large  arrival  and  service 
rates  imply  larger  shifts  in  state.   This  means  having  large 
holding  costs  and  soon  after  no  holding  costs  and  conversely. 
This  implies  many  set-up  and  shut-dov/ns  and  hence  i*  is  very 
low  so  that  the  server  would  only  be  shut  down  when  empty. 


CHAPTER  IV 

MULTI- SERVER  MODEL 

Introduction 

Let  us  return  now  to  the  more  general  problem,  that  of 
finding  optimal  policies  for  the  operation  of  a  service  mecha- 
nism with  an  arbitrary  number  of  servers. 

The  assumptions  made  at  the  beginning  of  Chapter  II  will 
still  hold,  except  the  following  change: 

(I')   Assume  there  are  s  servers.   Their  service  times 
are  independent  exponential  random  variables  each  with 
rate  y.   Customers  are  still  processed  in  the  order  of 
their  arrival.   If  a  server  is  closed  down  whil  proc- 
essing a  customer,  that  customer  rejoins  the  queue. 
Also,  just  as  in  the  single  server  case,  we  shall  assume 
that  for  k,  the  number  of  open  servers,  our  arrival  rate  is 
such,  that  given  any  £>0,  there  exists  and  M(e)  such  that: 

P[X^>M(£) ]<£ 

This  along  with  the  assumption  that  sA<y  will  guarantee  that 
decisions  will  only  have  to  be  made  in  a  finite  number  of  states, 

The  cost  structure  is  affected  only  in  that  there  is  nov/ 
an  operating  cost,  set-up  cost  and  shut-down  cost  for  each 
server,  respectively  C,  A,  and  B.   The  transition  probabilities 
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will  now  be  denoted  as 

p..  =  Pr{X   ,  =  j |x  =  i  and  k(=0,l,...,  s)servers  are  opened} 

where  X   is  again  the  niimber  of  customers  in  the  system  at 
time  t. 


Form  of  the  Optimal  Policy 

In  this  section  we  will  describe  a  policy  which  we  will 
define  to  be  optimal.   This  policy  most  closely  resembles  that 
of  the  (j*/  i*)  for  the  single  server. 

Given  the  state  of  the  system,  we  are  interested  in  deter- 
mining unique  numbers  which  tell  us  how  many  servers  will  be 
open  after  a  decision  is  made.   Notationally ,  let  us  describe 
an  optimal  policy  as  follows: 

.*.*       .*     .*         .* 
(i,k  :  3  Q  /  j  2^  /  .  •  •  /  J]^_]^'  ^k+l'"'"'  ^s 

where  the  i,  k  represents  the  present  state  of  the  system, 

i.e.,  i  customers  present  and  k  servers  opened.   The  sequences 

*        * 

{j  }  and  {i„}  are  nondecreasing  sequences  with  the  following 

interpretation : 

*  * 

If   i      ,<i<i      and  k<m-l,    open  m-l-k   servers    (have  m-1   open 
m-1—       m  —  ^  ^ 

*  * 

after  decision).   If  i   i<i<i   and  k>mc,  close  k-m  servers 

-^m-1   —  m      — 

*       * 

(have  m  open  after  the  decision).   If  i>_i   or  i<_j^,  have 

all  the  servers  open  or  closed  respectively.   If  i  does 

not  meet  any  of  these  conditions  keep  k  servers  open. 


,  Note  how  this  conforms  to  the  original  form  for  an  optimal 
policy  described  in  Chapter  I. 
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Figure  3:   Costs  associated  with  a  multi- 


server  queue. 
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An  optimal  policy  may  permit  going  from  j  to  j+1  or  j  to 
j-1  for  all  j.   If  this  is  the  case  both  sequences  are  increas- 
ing.  However,  an  optimal  policy  may  dictate  a  state  £  where 
the  decision  to  open  only  one  server  will  never  be  made.   In 
this  case  i^_^^   =   i^  (or,  possibly  j  ^_-^   =   j^).   This  is  best 
illustrated  by  a  simple  example. 

Let  us  assume  that  s=3  and  presently  i=0.   Figure  3  depicts 
two  possibilities  as  to  what  the  cost  might  be  as  a  function 
of  the  number  of  servers  to  be  opened. 

In  (a),  i^,  i^  and  i^  are  well  defined  to  be  the  crossing 
points  of  the  (0,1),  (1,2)  and  (2,3)  intersections  respectively; 
and  the  sequence  of  i's  is  increasing.   In  (b)  it  is  never  opti- 
mal to  have  one  server  open  so  in  order  to  conform  to  our  rules 
our  optimal  policy  will  have  i,  =  i„  and  the  sequence  of  i ' s 
is  nondecreasing. 


Dynamic  Programming  Formulation 

Now  that  the  elements  of  the  model  are  given  let  us  again 
set  up  the  dynamic  programming  formulation  for  the  finite  hori- 
zon. 

Define   ()>  (i,a,k)  as  the  minimum  expected  cost  with  i 
customers  present,  k  servers  opened  with  n  periods  remaining 
in  our  horizon, using  discount  factor  a. 
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Then  our  recursive  relationships  are: 


(t)^(i,a,k)  =  min 


K(i)+kB       +a  Z  p"  4)^_^(j,a,0) 
K(i)  +  (k-l)B+C+a  Z  Pi-i^n-l^^'"''-'-^ 

•  * 

K(i)+kC       +a  Z  p^^^_^(j,a,k) 

k+1 


K(i)+(k+l)C+A+a  Z  p  .  (^   ,(j,a,k+l) 

in   n-i 

_K(i)+sC+(s-k)A+a  Z  pJ.<t>^_-L{j,ct,s) 


(2) 


where 


(^Q(i,a,k)  =  K(i) 

and  as  an  analogous  assumption  from  the  single  channel  system 
we  can  say  that  there  exist  numbers  Mq ,  M^ ,  ...  M   ,  which 


demand  that 


k+1 


(|)  (m,ct,k)=K(m)  + (k+l)C+A+a  Z  p^  .  4)^_,  ( j  ,  a,k+l)  for  m>iM,   (3) 

where  there  is  a  one-to-one  correspondence  between  the  k  and 
M,  .   Since  this  sequence  (Mt^)  is  finite  there  is  certainly  a 
maximum  and  for  simplicity  we  will  impose  (3)  for  m  >_  max  {M,  }=M. 
All  this  says  that  whenever  M  customers  are  reached  and  k  ser- 
vers are  opened  we  will  automatically  open  one  more  server. 

Furthermore,  the  assumptions  that  K(i)  is  nondecreasing 
and  convex  will  again  be  made,  along  with  assum.ing  that  for 
each  nondecreasing  function  h(j) 


SPi^   h(j) 
j  =  0  ^^ 


is  nondecreasing  in  i  for  each  k. 
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Proof  for  Finite  Horizon 

The  same  sequence  of  developing  is  used  for  the  multi- 
server  system  as  was  used  for  single  server.   In  this  chapter 
only  this  logical  sequence  of  steps  will  be  mentioned  to  ensure 
our  results  for  an  arbitrary  number  of  servers.   Proofs  for 
the  theorems  will  appear  in  Appendix  2. 

For  n=l,  let  us  illustrate  how  the  result  carries  by 
considering  s=2.   Then, 

K(i)      +a  E  p°j  K(j) 


i{i,(i,a,0)  =  min  /  K(i)+A+C  +a  Z  p^  .  K(j)    > 


K(i)+2(A+C)+a  E  p^.  K(j) 


K(i)  +  B  +a  Z  p^.  K(j) 


J 


41  (i,a,l)  =  min  /  K(i)+C+a  E  p]^  .  K(j) 


K(i)+A+2C+a  E  p  .  K(j) 


/ 


K(i)  +  2B  +  a  E  p^.  K(j) 


(J),(i,a,2)  =  min  /  K  (i)+B+C+a  E  p^  .  K(j) 


K(i)+2C  +a  E  p^   K(j) 


; 


when  the  server  is  closed  it  must  be  shown  that 

0      1 
E  K(j)  [p .  .  -  p.  •]  is  nondecreasmg 


and  that 


1      2 
E  K(j)  [pt .  -  p.  .]  is  nondecreasmg 
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(The  sum  of  these  two  is  just 


I  K(j)  [p°.  -  o^.] 


which  would  certainly  be  nondecreasing  as  it  is  the  sum  of 
two  nondecreasing  functions.)  Doing  this  leads  to  proving 
Theorem  1:   For  n=l  the  policy  of  form  (1)  is  optimal. 

To  extend  the  results  for  arbitrary  n,  let  us  point  out 
the  steps  that  make   the  analysis  for  the  single  server  and 
that  for  an  arbitrary  number  of  servers  similar. 

For  n=2  it  becomes  necessary  to  show  that 

^1  ^2 

()>,  (i+x  -   r  y.  ,a,k,)  -  (|)-,(i+x  -   Z  y.  ,a,k^) 

^  i=0   ^     -^      ^  i=0   ^     "^ 

for  each  k,  and  k„, such  that  k-  >  k, ,  is  a  nondecreasing  func- 
tion of  i.   This  is  equivalent  to  saying  that  there  is  at  most 
one  crossing  point  for  any  two  alternatives  and  since  the  curves 
are  nondecreasing  (1)  will  be  ensured  optimal.   The  proof  for 
this  is  the  proof  of  Theorem  2  in  the  appendix  and  is  analagous 
to  Theorem  2  of  Chapter  II. 

Again,  continuing  with  the  induction  hypothesis,  i.e., 
assuming  <}>  (i,a,k),  n=3,4,...,  k  has  an  optimal  rule  given 
by  (1) ,  the  enumeration  procedure  becomes  more  difficult  but 
once  again  Theorem  2  of  Chapter  II  follows  in  precisely  the 
same  manner  (this  once  again  is  so  because  alternatives  only 

have  to  be  compared  two  at  a  time) .   This  is  done  first,  by 

*     * 

providing  the  analog  of  Lemma  3  (proving  ij^ ,  n^ji^.-i  )  /  and 

secondly,  by  using  the  transitive  relations  that  if  a-b  is 
nondecreasing  and  b-c  is  nondecreasing,  then  a-c  is  nondecreas- 
ing (a-c  =(a-b)+ (b-c) ) .   The  details  are  in  Appendix  II. 
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Also,  the  discussion  on  piecewise  convexity  is  simply 
extended.   The  definition  allows  a  function  to  be  the  smallest 
of  a  denumerable  number  of  convex  functions  and,  hence,  the 

addition  of  the  extra  alternatives  still  permits  ii     (i,a,k) 

c  n 

to  be  piecewise  convex.   Also,  the  argument  related  to  the 
terminal  points  holds  since  having  the  possibility  of  s 
servers  just  allows  for  the  possibility  of  s  different  weights. 

This  completes  the  discussion  for  the  finite  horizon  and 
we  have  shown  that 

Theorem  2:   For  an  arbitrary  nuniber  of  servers  our  optimal  policy 
for  the  cost  functions  given  in  (2)  is  satisfied  by  (1). 


Infinite  Horizon  and  Averaging 

Proceeding  as  before  we  shall  prove  that  (1)  is  the  opti- 
mal policy  for  the  infinite  horizon,  i.e.,  when 


K(i)  +  BK     +a  E  p"  .  !t)  (j  ,a,0) 


K(i)+B(k-l)+C+a  E  p  .  (j)(j,a,l) 

•  • 

K(i)  +  kC  +aZp^.  (|)(j,a,k)  (4) 

K(i)  +  (k+l)C+A+a  Z  p^"!" ■'■<*' (j /Ct,k+1) 

13 


(})(i,a,k)  = 


_K(i)  +  sC  +  (s-k)A+a  Z  p'^  .  *(j,ct,s), 
Again  the  proof  will  be  done  for  only  one  state  (i,k)  but  by 
its  nature  is  seen  to  hold  for  all  states.   The  proof  is 
Theorem  3  in  Appendix  II. 

Moving  into  the  discussion  on  the  averaging  criterion, 
the  assumption  that  for  each  number  of  open  servers  there  are 
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only  a  finite  niiii±ier  of  decision  points  we  are  able  to  prove 
that  (1)  is  the  optiual  policy  as  shown  in  Theorem  4  from 
Chapter  II.   Theorem  4  in  Appendix  II  gives  this  proof  implying 
that  (1)  is  the  optimal  rule  for  4)(i,k)  where 

<|)(i,k)  =  lim   (1-a  )(i)(i,a  ,k)  where 

1  ^ 
(})(i,k)  =  lim  ^   Z  g(i,k)  and  g  (x  ,k)is  the  one  step  loss 

function. 

Concluding  Remarks 

Certainly,  the  extension  of  Theorem  5  in  Chapter  2  holds 
for  the  multi-server  model.   That  is,  we  will  never  close  all 
the  servers  with  customers  present,  except  possibly  when  there 
is  a  finite  horizon. 

In  essence,  therefore,  this  chapter  along  with  Appendix  II 
shows  that  the  multi-server  and  single  server  model  are  very 
similar,  in  the  sense  of  choosing  optimal  policies.   This  is, 
however,  an  exception  rather  than  the  rule  in  dynamic  program- 
ming formulations.   Bellman  points  out  that  under  very  general 
conditions  when  there  are  only  two  possible  decisions  that 
there  is  some  point  which  uniquely  separates  our  decision  region. 

However,  for  more  than  two  decisions  he  maintains  that  in  general 

[2] 
our  decision  regions  are  not  so  divided. 


[21 

Bellman,  R. ,  op.cit. ,  page  75 


CHAPTER  V 


CONCLUSION  AND  EXTENSIONS 


The  purpose  of  this  paper  was  to  ascertain  optimal  poli- 
cies for  queuing  systems.   The  form  of  the  optimal  policy  for 
the  single  server  had  been  found  before,  but  only  when  obser- 
vations were  at  the  end  of  arrival  and  departure  epochs,  rather 
than  having  observations  in  equal  time  intervals.   Also,  the 
only  work  previously  done  in  existence  of  optimal  policies 
for  multi-server  models  was  a  tv;o-server  case  discussed  by 
Heyman.   In  this  discussion,  he  always  had  one  server  open 
and  the  policy  was  to  determine  when  to  open  the  other.  Vie 
allow  for  any  finite  number  of  servers.   We  are  able  to  con- 
clude that  the  numier  of  servers  open  at  some  point  in  time 
is  given  by  an  intuitively  appealing  rule.   This  rule  is 
analagous  to  the  (s,S)  policy  in  inventory  theory. 

The  applications  of  the  work  in  this  paper  are  quite  wide- 
spread.  In  any  situation  where  there  is   a  variable  number 
of  service  stations  which  may  be  open,  one  would  know  how  many 
servers  should  be  opened  or  closed.   More  might  be  learned 
by  observing  additional  computational  results,  as  the  purpose 
of  this  paper  is  not  to  find  optimal  policies  but  only  the 
form  at  v/hich  they  take  on.   Sobel  points  out  many  specific 
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problems  relating  optimal  policies  for  queuing  system.s  to 
inventory  and  replacement  theory. 

There  are  several  extensions  to  be  seen  from  the  work 
done  in  this  paper.   Certainly  there  is  a  need  for  greater 
com.putational  facility  in  obtaining  the  optimal  policy  for 
a  given  model.   At  present  the  closest  computational  experi- 
ence for  the  multi-server  model  is  in  a  paper  by  Moder  and 

12 
Phillips.     Also,  there  is  no  reason  why  Howard's  scheme 

cannot  be  extended  to  deal  with  this  case.   Computational 
experience  is  very  important  for  a  problem  of  this  type  as 
its  application  will  only  become  meaningful  when  the  param- 
eters which  we  might  be  free  to  choose  are  set  at  the  proper 
levels.   That  is  to  say  that  even  though  our  optimal  policy 
is  determined  only  from  the  number  in  the  system  and  the 
nximber  of  servers  open,  there  may  be  other  param.eters,  such 
as  service  rate,  which  might  substantially  affect  our  optimal 
policy. 

The  main  restriction  that  had  to  be  m.ade ,  so  that  periodic 
review  can  be  used,  was  that  our  arrival  and  service  distribu- 
tions were  exponential.   As  was  mentioned  this  assumption  could 
be  dropped  if  we  are  v/illing  to  increase  our  state  space. 
That  is,  the  state  would  be  not  just  a  number  in  the  system 
and  the  number  of  open  servers  but  also  the  time  from  the  last 
arrival,  if  arrivals  were  to  become  general.   Investigation  of 
this  problem  is  to  be  considered.   Computationally,  one  would 
almost  certainly  have  to  use  Jewell's  Markov-renewal  technique. 

Another- extension  is  to  consider  finding  optimal  policies 
when  there  are  several  machines  with  different  service  rates. 
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This  was  avoided  because  one  must  either  make  a  predetermined 
rule,  or  include  the  finding  of  the  rule  as  part  of  finding 
the  optimal  policy  to  determine  which  of  the  available  machines 
an  arriving  customer  will  be  serviced  on.   When  this  is  done 
one  can  introduce  switching  type  policies  as  alternatives, 
i.e.,  if  a  server  is  closed  down  while  serving  a  customer, 
the  customer  may  immediately  switch  to  another  machine.   Heyman 
discusses  rules  of  this  type. 

A  slightly  different  but  very  interesting  problem  can  be 
thought  of  as  an  extension  is  that  of  finding  the  optimal 
policy  as  a  function  of  some  other  parameter.   In  particular, 
finding  an  optimal  rule  as  a  function  of  the  arrival  rate, 
rather  than  of  the  number  in  the  system,  may  have  useful  appli- 
cations.  If  one  could  check  on  the  rate  at  which  customers 
entered  a  supermarket,  for  example,  it  might  be  possible  to 
predict  when  servers  should  be  opened  or  closed  so  as  to  avoid 
idle  time  and  long  queues. 

Of  course,  balking  or  reneging  can  also  be  introduced  to 
the  system.   In  connection  with  this  we  could  consider  a  modi- 
fied loss  system  which  has  application  to  telephone  switchboards 
That  is,  consider  a  customer  who  tries  to  enter  the  system  and 
is  turned  away  (placing  a  call  and  finding  all  circuits  busy) 
but  will  return  with  a  certain  probability.   Associating  a 
cost  with  a  lost  customer  it  is  to  be  determined  how  many  ser- 
vers should  be  kept  open  so  as  to  minimize  cost. 

■  Another  change  in  the  model  would  be  to  change  the  queue 
discipline;  introduction  of  priorities  may  certainly  make  this 
work  applicable  to  time-shared  corriputer  systems. 


APPENDIX  I 


PIECEWISE  CONVEXITY 
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The  following  is  due  to  Zangwill 

Definition:   Piecewise  convex  function  (PCF)  --  A  function 

f  (Z)  is  said  to  be  piecewise  convex  if  there  is  a 

sequence  of  convex  functions  {g. (Z)}  such  that 

f  (Z)  =  inf  {g^(Z)  } 
i 

Definition:   Let  f(Z)  =  inf  {g.(Z)}  be  PCF.   Define  a  basic 

i 

set  B.  as 
1 

B^  =  {Z|  f  (Z)  =  g^(Z)  } 

Definition:   Let  the  terminal  set  T  be  defined  as 

T  =  {Z|  f(Z)  =  g^(Z)  =  gj^(Z)   for  i  ^^  k} 

The  next  lemma  is  an  important  one  in  the  concept  of  PCF. 
Lemma  1:   Let  {f . (Z)}  be  a  sequence  of  PCF  each  with  respec- 
tive terminal  set  T..   Then  F(Z)  =  iw^f^(Z),  with 

i 
w.  >  0  is  PCF  with  terminal  set  T  =  U  T.. 

i   ^ 
Proof:   The  proof  is  by  induction  on  the  number  of  functions 

For  one, the  result  is  clearly  true.   Assume  it  is  true 

for  k,  i.e., 

k  ^   k 

F,  (Z)  =   I  w.f . (Z)  is  PCF  with  terminal  set  T  =   U   T. 

^      .  ,   ^  ^  i=l 

1=1 
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By  the  definition  of  PCF 
F,  (Z)  =  inf  {ghz)  } 

for  some  sequence  of  convex  functions. 

2 
Let  W],j.i  f],j.i  (2)  =  inf{g.(Z)}  for  a  sequence  of  convex 

functions.   Then 

^k+l^^^^^k^^^+^'k+l^k+l^^^"  inf{g];(Z)}+inf{g^(Z)} 

i  J    -^ 

=  inf{g.^(Z)  +  g?(Z)} 
i/j  ■' 

1         2 

But  g.  (Z)  +  g.  (Z)  is  convex  for  all  i  and  j  and  hence 

Fj^_j,^(Z)  is  PCF. 

1        2 
Now,  a  point  h  is  in  the  terminal  set  of  g  (•)  +  g  (•)  if 

and  only  if  g-""  (h)  +  g^  (h)  =  g.  (h)+g^.  (h)=g-!-  (h)  and 

^1      ^1     ^2 

either  (a)  ij^7^i2  ^^'^   ^1~^2'  ^^^  ■^1~''^2  ^^^  ^i^^2'  °"^ 
(c)  i-iT^i-)  and  j^T^j^.   If  (a)  occurs  then  h  e  T  ,  if  (b) 
occurs  then  h  e  T,  ,  and  if  (c)  occurs  then  h  e  T  HT,  ,  . 

Also,  if  h  £  T  ,  then  either  (a)  or  (c)  occurs;  if  h  e  "^u+i' 

k+1 
then  either  (b)  or  (c)  occurs  and  h  e  T    .   Another 

useful  lemma  is, 
Lemma  2:   If  g(Z)  and  h(Z)  are  PCF  then  f(Z)  =  min  [g(Z),h(Z)] 

is  PCF. 

Proof:   By  definition  of  PCF 

g(Z)  =  inf  {g.  (Z)  }  and  h(Z)  =  inf  {h.  (Z)  } 
i    ^  j     ^ 

for  some  sequence  g  and  h. 
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Hence, 


f (Z)  =  min  [inf  {g.  (Z)  },  inf  (h.  (Z) }] 
i    ^       j    ^ 


=  min  [inf  {g^(Z)  ,  h.  (Z)}] 


=  inf  {g. (Z),  h. (Z)} 


and  f(Z)  is  PCF. 

00  00 

Lemma  3:    Z  p..K(j)  and   Zq..K(j)  are  convex. 
j=0   ^^         j=0  ^^  ^  ~ 

Proof:   (Two  ways  are  illustrated  --  one  for  I   p. .K(j)  and 

the  other  for  i:g..K(j)) 

eo 

^  PhsK(J)  =  E^  K(i+x) 
j  =  0   ^-'         ^ 

=  Z  P^  (x  arrivals  in  time  period)  K(i+x) 

X 

which  is  a  linear  combination  of  convex  functions  and 
hence  is  convex. 

Or  as  an  alternative, 
SK  (i^+x-y )  +  (1-9  )  K  (i2+x-y)  >.K  (ei^+  (1-9  )  i^+x-y) 

eE^EyK(i^+x-y)+(l-9)E^E^K(i2+x-y)>.E^E^K(9i^+(i-9)i2+x-y) 
and  hence 


I  g.  .  K(j)  =  E  E   K(i+x-y) 

j=o  ^^  ^  y 


IS  convex. 
Recalling  that. 


0,.  f  N 

<j>,  (i,a)  =  min     K(i)  +   aZ  p..  K(j) 

J  j=0   ^3 

I  N 


K(i)  +  A  +  C  +  a  I   q.  .  K(j) 


j  =  0 


ID 


59 

and  using  the  definition  of  PCF  we  conclude  that  (}),  is  PCF. 
Next,  since 

00 

I  p .  .  (|)  (j,a)  =  E  4),  (i+x,a)  =  Z  Pr  (x  arrivals)  <}>,  (i+x,a) 
j  =  0   ^3   -L         XI  1 

and 

oo 

I  q    *°(j,a)  =  E  E   (j)°  (i+x-y ,  a ) 
j  =  0   -^J   -^  X  y   J. 

=  E  I  Pr  (x  arrivals)  Pr  (y  services)  4>-i  (i+x-y,  a) 

along  with  Lemma  1  enables  us  to  say  that  Z  p.  .  4)^  (j,a)  and 

Z  q.  .  (})^  (j,a)  are  PCF.   Since  the  addition  of  a  convex  function 

to  a  piecewise  convex  function  is  obviously  piecewise  convex, 

i^y{i,a)    is  a  PCF,  a  result  achieved  from  Lemma  2. 

Finally,  alternate  use  of  Lemmas  1  and  2  leads  to  the 

following: 

Lemma  4:   4;  (i,a)  for  all  n  >  1  is  PCF. 
n  — 

The  result  also  follows  immediately  for  cf)  (i,a). 

n 

Clearly,  since  the  set  of  terminal  points  does  not  depend 
on  the  weights, 

^Pi+x,j  *k-l  ^^'"^  ^^^  'Pi+x-y,j  *k-l  ^^'""^ 
have  the  same  terminal  points.   Also,  each  interval  between  any 
two  terminal  points  of  a  piecewise  convex  function  is  convex. 
Hence,  to  show 

^^Pi+x,j  *k-l(^'"^  -  Pi+x-y,j  *k-l  ^"-5'^^^ 
is  nondecreasing  between  terminal  points  one  can  resort  to  the 
discussion  used  to  show  the  same  thing  for 
^  IPij  -  qij]  K(j). 
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Repeated  use  of  the  fact  that  the  terminal  points  are 

the  same  and  that  the  expression  is  nondecreasing  between 

these  points  enables  us  to  conclude. 

Lemma  5:   ^  [p^  ,^  ^  -  P,-  .^  „  .;]  *i-_i  (j/^)  is  nondecreasing 
i+x  f J    x+x  y , J    K  X 

in  i.   Most  certainly  the  result  also  holds  for 


^  t^i+x,j  -  ^i+x-y,j]  *J-1  ^^'^^ 


APPENDIX  II 


EXTENSIONS  OF  TtlEOREMS  TO  MULTI-SERVER  QUEUE 


Lemma:    Z  [p . .  -  p . . ]   K(j)  is  nondecreasing  for  m<n. 
j=0   ^^     ^^ 

Proof:   Just  as    I  q . .  K(i)  =  E  E   K(i+x-y), 

m 
Ip..K(j)=EE   ...E    K(i+X"   I        y .) 

m 
Letting    E  y.  =  Y  ,  from  Chapter  II  we  know  that 
j  =  0  ^     "* 

K(i+x-Y  )  -  K{i+x-y  )  for  Y  >Y   is  nondecreasing 
m  n       n  m 

in  i.   Hence,  so  would  the  difference  in  their  expecta- 
tions be  and 

OO 

Z   [p.^  -  p.^]  K(j) 
j  =  0   ^^    ^^ 

for  all  m<n  is  nondecreasing. 

Theorem  1:   For  n=l  the  optimal  policy  is  of  form  (1)  in  Chap- 
ter IV. 

Proof:   Let  k  be  the  number  of  open  servers. 
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Then, 


()),  (i,k,a)  =  min 


K(i)+Bk 
K(i)+B(k-1)+ 
K(i)  + 
K(i)+A 


+  aEp^j   K(j) 


C+a  Z  p.  .   K(j) 


kC+a  E  p.    K(j) 


+  (k+l)C+aj:  p^;^^  K(j) 


K(i)+A(s-k)+ 


aipr:   K(j) 


The  policy  (1)  implies  that  there  exist  unique  numbers 
which  determines  exactly  how  many  servers  should  be  open 
with  i  customers  in  the  system,  given  k  were  open.   Now, 
for  any  two  alternatives  m  and  n,  m<n. 

K(i)  +  (Constant)  +  a  I  p .^  K ( j )  (^) 

K(i)  +  (Constant)'  +aZp.^  K(j) 

or 
2  [p^j  -  P^jJ   K(j)  =  Constant 

can  occur  for  at  most  one  value  of  i.   Since  the  left 
side  is  nondecreasing  from  the  preceding  lemma,  once 
alternative  n  is  chosen,  alternative  m  can  never  be 
chosen  again  for  all  m<n.   Hence,  the  sequences  are 
nondecreasing,  which  determines  v/hen  an  alternative  should 

be  chosen. 

^1  ^2 

Lemma:   E  {p.  .   ^,  (j,a,k,)  -  p-  ■  •P-,     (j,a,k-)  is  nondecreasing 

for  k^<k2 
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Proof:   This  can  be  written  as 


E[(f.^(i+x-Yj^  ,a,k^)  -  <J)j^  (i+x-Yj^  ,a,k2)] 

where  the  expectation  is  taken  over  random  variables  x, 
Y,  ,  Y   .   If  the  bracket  is  nondecreasing  for  all  i  then 

certainly  the  expectation  will  be.   So  let  us  consider 
<()j^  (i+x-Yj^  ,a,k^)  -  ((!^(i+x-Yj^  ,a,k2) 

Now  for  any  i , 

(j.^  (i+x-Yj^  ,a,k^O  =  K(i+x-Y,  )  +  (Constant) 


+  «^Pi+x-Y,   '^(3) 


The  difference  is  nondecreasing  in  i  since  K(i+x-Y,  )  - 

^1 

K(i+x-Y,  )  is  nondecreasing  and  the  difference  of  the 

summations  is  by  the  first  lemma  in  the  appendix. 
Assuming  that  (|>  (i,a,k)  n=2,3,4,...r  has  an  optimal  rule 
given  by  (1)  and  that 

^1  ^2 

^  ^^ij   *^n  ^li'^'^i^    ~   Pij  ^xi    (J'Ct/k^O]  for  k^  <  k^ 

is  nondecreasing  in  i ,  (1)  must  be  shov/n  optimal  for  n=r+l. 


Theorem  2:   (1)  is  optimal  for  (j) 


r+1 


Proof:   Since 


^\ 


^r+l  (i'"'k)  =   \ 


K(i)+kB 
K(i)  + 


-a  T.   p^  .  (}!j.  (1  ,a  , 


0) 


kC+a  Z  p.  .  <j)  {i,a,k) 


K(i)+ (s-k)A+sC+a  I  p.  .  <p     (i,a,s) 


for  optimal  solution  to  be  (1)  it  must  again  be  true 
for  alternatives  m  and  n,  m  <  n,  the  difference  must  be 
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nondecreasing.   This  will  guarantee  that  the  sequences 
{j*}  and  {i*}  will  be  nondecreasing.   Since  this  difference 
is 

it  is  nondecreasing  by  the  induction  assumption. 
Theorem  3:   <J)(i,a,k),  as  defined  by  (4)  in  Chapter  IV,  has  an 

optimal  policy  given  by  (1)  in  Chapter  IV. 

Proof:   (Done  for  an  arbitrary  state  (i,k)) 

Define  T.((^   ,  )  as  the  cost  associated  with  the  j    alterna- 
j      n-i  -' 

tive  j=0,l,...,s.   For  example, 

"^O^^n-l^  "        ^^^^  "^  ^^  ■^''^Pij  ^n-l^^''''^^ 
Then, 

(j>  (i,a,k)  =  min       T.  (<*i„_t  ) 
j=0,...,s   ^   ^  ^ 

Let  j  (n)  =  index  which  gives  the  minimum  of  T .  (<))),  j  =  0,...,s 
This  implies  that 

and 

*^^,(i,a,k)  =  T.(^^(*J  ^Tj(n-l)(V 

Subtracting, 

U^(i,a,k)-*^^^(i,a,k)|^max  [lT.(^_^j{*^_^)-T.(^_^)(*^)|, 

l^j(n)(*n)-^j(n)(*n-l)|] 
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imax        [|T   (*^_,)  -  T  (^^)i] 
3  =  0,1,  .  .  .  ,s    -" 

<.max   riK(i)+kB+a  E  p^   'i'n-1  ^^ '"  '  °  ^  ~ 

(K(i)+kB+a  Z  p.  .  (})^(j  ,a  ,0)  )  1  ,  ... 

lK(i)+Ck+a E  p^^  ^^_^(j,a,k)  - 

(K(i)+Ck+a  E  p^^  <})^(j,a,k)  )  1  ,  ... 

lK(i)+sC+(s-k)A+a  E  p.^  '!'j^_]_(j '^'S)  - 

(K(i)+sC+(s-k)A+a  E  p^^  *^(j,a,s)) 1  J 
((J)   ,  (i  .a  .r 
£  a  E  p^^U  j,_i(J.a.k)  -  <!)^(j,a,k)  1 

Hence 

inaxU^(i,a,k)  -  (!>^^-,_  (i  ,a  ,k)  |  <. 
i 

k     I  I 

aEp..  max   <{)   ,(i,a,k)  -  <j)^(i,a,k)l 
xj  n-i  n 

<amaxl  ^   T(i,ci,k)  -  <i5„(i,a,k)l 

X 

Define,  U   =  max  [\  <^     (i,a,k)  -  *^_j_  (i  ,a  ,k)  1  ] 
i 

and  hence 

U   ,<  aU   <  a   U   T  . . . .<   a  U, 
n+1—   n  —     n-1     —     i 

and, since 

00  00  U, 

E    U   ,<     E   a%-  = 


imax     a]  E  p^^  ^*n-l  ^^/^fr)  -  4>^(j,a,r)) 
r 


n=0    ^-^1-  n=0     1    1-" 


we  conclude  that 


E  I  *  ^^(i,a,k)  -  ^^(i,a,k)  ] 
n=0 
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converges  uniformly  and  that 
*^(i,a,k)   ^({i(i,a,k) 

where  i>{i,a,k)    has  been  previously  defined. 
Theorem  4:   (1)  of  Chapter  IV  is  optimal  for  the  averaging 
cost  criterion. 


*    * 


*       * 


a 


Proof:   Let  (j^,  j^,  ...  j^_^,  i^_^^,  ...  i^)^   =  p' 

be  the  optimal  policy  for  (t>(i,a,k).   Let  {a  },  with 
lim  a  =1  be  a  sequence  such  that  P   =  P  for  all  t. 

Since  the  number  of  possible  policies  is  finite  such  a 
sequence  and,  therefore,  P*  must  exist.   Let  P  be  any 
other  policy.   Then  certainly 
<j)p(i,a^,k)^   (f)(i,a^,k) 

Hence, 

lim   (l-a  )  <f  (i,a  ,k)  ^  lim  (1-a.)  Mi,ct^,k) 
t^"  ^      t-^<=° 

and  again  using  the  result  that 
'f(i,k)  =  lim  (l-a  )  (f>(i,a  ,k) 

where  a{i,k)  has  been  previously  defined,  it  is  the  mini- 
mum averaging  cost. 
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